Skip to content

feat: Add usage budget controls#2270

Open
niemyjski wants to merge 13 commits into
mainfrom
niemyjski/add-usage-budget-controls
Open

feat: Add usage budget controls#2270
niemyjski wants to merge 13 commits into
mainfrom
niemyjski/add-usage-budget-controls

Conversation

@niemyjski

@niemyjski niemyjski commented May 31, 2026

Copy link
Copy Markdown
Member

Summary

Adds production-hardened organization budget alerts, explicit project ingest limits, and automatic noisy-project sampling. The branch is merged forward to current origin/main, including the organization icon UI and controller coverage.

User-visible behavior

  • Organization Usage has a dedicated card for enabling, disabling, clearing, and previewing 1–99% budget alerts.
  • Project Usage has a dedicated card for fixed or percentage limits, effective-limit previews, clear support, and smart-throttle status.
  • Project charts prefer the effective project limit and identify whether the line is the project or organization limit.
  • Budget-alert and smart-throttle emails report current usage and the applicable threshold or fair-share limit.

Safety/compatibility

  • Existing organization overage, authorization, status-code, public API, and WebSocket behavior is preserved; new contract fields are additive.
  • JSON null clears only the two new settings objects; shared PATCH semantics are unchanged.
  • Unlimited/uncapped ingestion returns without cache or repository reads. Finite traffic uses one bulk counter read, and project-list throttle state uses one bulk cache read.
  • Smart throttling is transition-based, uses stable 5% sampling (including single-event posts), and counts each rejected event once as blocked.
  • Alert/throttle claim and publication failures are isolated from ingestion. Delayed handlers recheck current settings, plan, usage, and throttle state.
  • No Elasticsearch mapping change or reindex is required.
  • Microsoft.OpenApi is pinned to 2.7.5 to clear GHSA-v5pm-xwqc-g5wc / CVE-2026-49451. dotnet list package --vulnerable --include-transitive reports no vulnerable NuGet packages.
  • Frontend package manifests and lockfile are unchanged from main; the existing static-build toolchain advisories reported by npm audit --omit=dev are not introduced by this PR.

Validation

  • dotnet build --no-restore: passed with 0 warnings and 0 errors.
  • Full backend suite, serialized against isolated Elasticsearch: 2,344 total; 2,342 passed; 0 failed; 2 intentional performance-tool skips.
  • Focused usage, notification-handler, event-post, organization/project controller, middleware, Delta, OpenAPI, and mailer suites: passed.
  • npm ci: passed.
  • npm run check: passed with 0 errors and 0 warnings.
  • npm run lint: passed.
  • npm run test:unit: 22 files and 299 tests passed.
  • npm run build: passed with the static adapter.
  • openspec validate --all --strict --no-interactive: passed.
  • git diff --check: passed.
  • Local Aspire health: http://localhost:7110/api/v2/about returned 200.
  • Local browser QA: organization/project save, validation, clearing, chart-limit source, dark mode, and 390 px responsive layout passed.
  • Local ingestion QA: a 2-event project cap accepted 2/5 and blocked 3/5; an uncapped control project accepted 5/5; clearing restored ingestion.
  • Local alert QA: one real threshold crossing produced exactly one email; a second post did not duplicate it; disabling produced no additional email.
  • Local smart-throttle QA: the noisy project activated at 5%, accepted 4/100, and blocked 96/100; the control project remained unthrottled at 3 accepted and 0 blocked.
  • Final thermo-nuclear review: no structural, hot-path, file-size, generated-artifact, or compatibility blockers.

Rollout/kill switch

EnableSmartProjectThrottling defaults to true and can be set to false through configuration/environment for an immediate operational rollback. Explicit project limits and organization budget alerts remain available when the automatic sampler is disabled.

OpenSpec

openspec/changes/add-usage-budget-controls is complete: all 39 tasks are checked and strict validation passes.

niemyjski and others added 5 commits May 30, 2026 23:36
Create proposal, design, tasks, and spec deltas for:
- Organization budget alert emails
- Automatic smart project throttling
- Optional project event budgets
- API compatibility
- Organizations/projects/users/auth behavior

Addresses customer feedback for budget alerts before overage
and issue #112 requesting smarter project-level throttling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add OrganizationBudgetAlertSettings model with threshold validation
- Add BudgetAlertSettings property to Organization model
- Add ProjectIngestLimit model with Fixed and Percentage types
- Add IngestLimit property to Project model
- Create UpdateOrganization DTO for organization updates
- Add BudgetAlertSettings to ViewOrganization
- Add IngestLimit, EffectiveIngestLimit, IsSmartThrottled,
  SmartThrottleSampleRate to ViewProject
- Update OrganizationController to use UpdateOrganization
- Update Mapperly mappers for new properties

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add OrganizationBudgetAlert and ProjectSmartThrottleApplied messages
- Add work items and handlers for budget alerts and smart throttle
- Add CheckBudgetAlertThresholdsAsync to UsageService for threshold-based
  email alerts with per-billing-period deduplication
- Add GetEventIngestAllowanceAsync for project-specific budget enforcement
- Add GetSmartThrottleRateAsync for fair-share-based sampling
- Integrate smart throttling and project budget limits into EventPostsJob
- Add IMailer methods and Mailer implementations for budget alert and
  throttle notification emails
- Add HTML email templates for both notification types
- Register startup actions and work item handlers in bootstrappers
- Update NullMailer and CountingMailer test doubles

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wire up EffectiveIngestLimit, IsSmartThrottled, and SmartThrottleSampleRate
in ProjectController.AfterResultMapAsync using UsageService methods.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Regenerate openapi.json to include UpdateOrganization, ProjectIngestLimit,
OrganizationBudgetAlertSettings, and updated ViewProject/ViewOrganization
schemas.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread src/Exceptionless.Core/Models/Organization.cs Fixed
…hrottle

Bug #1 (regression): OrganizationController.CanUpdateAsync unconditionally
called IsOrganizationNameAvailableInternalAsync(changed.Name) without
checking if Name was included in the PATCH delta. When only
BudgetAlertSettings is sent, Name is null, causing the check to return
false and reject the PATCH with a spurious duplicate-name error.

Fix: Add ContainsChangedProperty(p => p.Name) guard, matching the
pattern already used in ProjectController.CanUpdateAsync.

Bug #2 (defensive): GetSmartThrottleRateAsync guarded maxEventsPerMonth < 0
but not == 0. If an organization is not found (cache miss + deleted org),
GetMaxEventsPerMonthAsync returns 0, which would cause division by zero.
In practice, line 508 (orgTotal <= 0) guard fires first for missing orgs,
but the fix makes the invariant explicit and prevents any future path that
reaches the division with maxEventsPerMonth=0.

Fix: Change guard from < 0 to <= 0.

Tests added:
- PatchAsync_OnlyBudgetAlertSettings_DoesNotRejectDueToNameValidation
  (fails without Bug #1 fix, passes with it)
- GetSmartThrottleRateAsync_OrganizationNotFound_ReturnsNoThrottle
  (defensive coverage for zero-limit path)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@niemyjski niemyjski marked this pull request as ready for review May 31, 2026 12:03
@niemyjski niemyjski requested a review from Copilot May 31, 2026 12:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds backend usage budget controls for organizations and projects, including budget alert settings, project ingest caps, smart throttling calculations, notification work items, and API contract updates.

Changes:

  • Adds organization budget alert and project ingest limit models/API fields.
  • Extends usage/event processing with alert checks, project budget allowance, and smart throttling sampling.
  • Adds notification messages, work items, mailer methods/templates, OpenSpec docs, OpenAPI baseline updates, and limited regression tests.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 19 comments.

Show a summary per file
File Description
tests/Exceptionless.Tests/Services/UsageServiceTests.cs Adds a regression test for missing organization smart throttle handling.
tests/Exceptionless.Tests/Mail/NullMailer.cs Adds no-op implementations for new mailer methods.
tests/Exceptionless.Tests/Mail/CountingMailer.cs Adds stub implementations for new mailer methods.
tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs Adds PATCH regression coverage for budget alert settings without name.
tests/Exceptionless.Tests/Controllers/Data/openapi.json Updates OpenAPI baseline for new usage budget fields and DTOs.
src/Exceptionless.Web/Models/Project/ViewProject.cs Exposes project ingest limit and smart throttle state.
src/Exceptionless.Web/Models/Project/UpdateProject.cs Allows project ingest limit updates.
src/Exceptionless.Web/Models/Organization/ViewOrganization.cs Exposes organization budget alert settings.
src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs Adds organization PATCH DTO.
src/Exceptionless.Web/Mapping/ProjectMapper.cs Ignores computed project usage budget fields in mapper.
src/Exceptionless.Web/Mapping/OrganizationMapper.cs Adds organization update mapping.
src/Exceptionless.Web/Controllers/ProjectController.cs Populates computed project ingest/throttle fields.
src/Exceptionless.Web/Controllers/OrganizationController.cs Switches update DTO and adjusts name validation condition.
src/Exceptionless.Web/Bootstrapper.cs Registers new startup notification subscribers.
src/Exceptionless.Core/Services/UsageService.cs Adds budget alert checks, project allowance calculation, and smart throttle calculation.
src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs Adds smart throttle notification work item.
src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs Adds budget alert notification work item.
src/Exceptionless.Core/Models/ProjectIngestLimit.cs Adds project ingest limit model and enum.
src/Exceptionless.Core/Models/Project.cs Persists optional project ingest limit.
src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs Adds organization budget alert settings model.
src/Exceptionless.Core/Models/Organization.cs Persists and validates organization budget alert settings.
src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs Adds smart throttle notification message.
src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs Adds budget alert notification message.
src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html Adds smart throttle email template.
src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html Adds budget alert email template.
src/Exceptionless.Core/Mail/Mailer.cs Implements new budget/throttle emails.
src/Exceptionless.Core/Mail/IMailer.cs Adds new budget/throttle mailer contracts.
src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs Adds smart throttle subscriber and email handler.
src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs Adds budget alert subscriber and email handler.
src/Exceptionless.Core/Jobs/EventPostsJob.cs Enforces project allowance and applies smart throttle sampling.
src/Exceptionless.Core/Exceptionless.Core.csproj Embeds new email templates.
src/Exceptionless.Core/Bootstrapper.cs Registers new work item handlers.
openspec/changes/add-usage-budget-controls/tasks.md Adds implementation/verification task plan.
openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md Adds auth/ownership requirements.
openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md Adds notification and queue requirements.
openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md Adds ingestion, throttling, and project budget requirements.
openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md Adds API compatibility requirements.
openspec/changes/add-usage-budget-controls/proposal.md Documents proposal and user-visible behavior.
openspec/changes/add-usage-budget-controls/design.md Documents detailed backend/UI design.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
Comment thread src/Exceptionless.Core/Jobs/EventPostsJob.cs Outdated
Comment thread src/Exceptionless.Core/Jobs/EventPostsJob.cs Outdated
Comment thread src/Exceptionless.Core/Jobs/EventPostsJob.cs Outdated
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
Comment thread src/Exceptionless.Core/Jobs/EventPostsJob.cs Outdated
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
niemyjski and others added 2 commits May 31, 2026 07:16
…t limits)

- Org manage page: budget alert settings section with threshold enable/disable,
  add/remove threshold chips, and save via patchOrganization
- Project manage page: event ingest limit section with type selector (none/fixed/percent),
  conditional numeric inputs, effective limit display, and smart throttle status indicator
- New schemas: BudgetAlertSettingsSchema, UpdateOrganizationSchema, IngestLimitSchema,
  UpdateProjectIngestLimitSchema
- New models: UpdateOrganization, OrganizationBudgetAlertSettingsUpdate, ProjectIngestLimitUpdate
- Regenerated API types from baseline OpenAPI spec (includes new budget/limit types)
- patchOrganization updated to use UpdateOrganization type for type safety

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bug fixes:
- Wrap CheckBudgetAlertThresholdsAsync in try/catch so budget alert
  failures never break event ingestion
- Use Math.Ceiling (not floor cast) for threshold event count and
  percentage project cap calculations
- Add RecordSmartThrottleAsync to publish deduped ProjectSmartThrottleApplied
  notification (once per billing period per project)
- Remove events.Count > 1 guard so single-event posts go through smart throttle
- Fix IsOrganizationNameAvailableInternalAsync to accept excludeOrganizationId
  so PATCH with current name + budget_alert_settings doesn't fail as duplicate
- Update OpenAPI baseline: budget_alert_settings and ingest_limit use proper
  $ref instead of untyped object schemas
- Fix CanRunJob_WhenSmartThrottleApplied test: org needs MaxEventsPerMonth
  set to a finite value and a second project for projectCount > 1 condition

Test coverage:
- UsageServiceTests: 36 new tests covering GetEventIngestAllowanceAsync,
  CheckBudgetAlertThresholdsAsync, GetSmartThrottleRateAsync, percentage
  limit ceiling, and budget alert edge cases
- ProjectControllerTests: ingest limit CRUD, validation, effective limit
  display, and smart throttle status
- EventPostJobTests: project-over-limit blocking and smart throttle
  discard with regression assertions
- OrganizationControllerTests: PATCH with same name + budget_alert_settings
  must not reject as duplicate name

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
Comment thread src/Exceptionless.Core/Services/UsageService.cs Fixed
niemyjski and others added 3 commits May 31, 2026 08:03
… validation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add precondition assertion to verify smart throttle is active before running
the job. Without warming the project-count cache first, the ES count query
for GetOrganizationProjectCountAsync may return stale results causing the
throttle to not fire. The pre-assertion also improves test documentation.

Also improve assertion error messages with diagnostic detail.
BLOCKER 1 (frontend): addThreshold() accepted val=100 but backend rejects
threshold >= 100 (valid range is 1-99). Fixed input max attribute and
validation error message in org manage page.

BLOCKER 2 (backend): non-atomic GetAsync+SetAsync for alertSentKey allowed
concurrent workers to both see key absent and both publish duplicate alerts.
Replaced with IncrementAsync (atomic); only the worker whose increment
returns 1 (first writer wins) sends the OrganizationBudgetAlert message.

Tests: adds regression test
CheckBudgetAlertThresholdsAsync_ConcurrentCalls_PublishesAlertExactlyOnce
which fires 5 concurrent IncrementTotalAsync calls and asserts exactly
1 alert is published.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 49 out of 52 changed files in this pull request and generated 14 comments.

Comment thread src/Exceptionless.Core/Jobs/EventPostsJob.cs Outdated
Comment thread src/Exceptionless.Core/Services/UsageService.cs Outdated
Comment thread tests/Exceptionless.Tests/Controllers/Data/openapi.json Outdated
Comment thread tests/Exceptionless.Tests/Controllers/Data/openapi.json
Comment thread src/Exceptionless.Core/Models/Organization.cs Outdated
Comment thread src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts Outdated
…ction-hardening

# Conflicts:
#	src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts
#	src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts
#	src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/manage/+page.svelte
#	src/Exceptionless.Web/Models/Project/UpdateProject.cs
#	tests/Exceptionless.Tests/Controllers/Data/openapi.json
#	tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.AppHost 39% 41% 136
Exceptionless.Core 72% 64% 8901
Exceptionless.Insulation 24% 23% 205
Exceptionless.Web 79% 67% 5247
Summary 73% (17596 / 24227) 65% (9002 / 13934) 14489

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants